DOCUMENT RESUME 



ED 463 413 



CE 082 983 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Ruhland, Sheila; Brewer, Jerrilyn 

Developing and Implementing Program Outcomes and Assessment 
Measures for Postsecondary Programs. 

2001 - 12-00 

13p . ; Paper presented at the Annual Meeting of the 
Association for Career and Technical Education (75th, New 
Orleans, LA, December 13-16, 2001). 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Alternative Assessment; * Col lege Outcomes Assessment; 
Communication Skills; *Competence; Competency Based 
Education; Curriculum Design; Education Work Relationship; 
Employer Attitudes; ^Evaluation Methods; Outcomes of 
Education; Program Development; Program Effectiveness; 
Program Validation; ^Student Evaluation; Teacher Attitudes; 
Two Year Colleges; ^Vocational Education; Writing Skills 
♦Career and Technical Education; ^Western Wisconsin 
Technical College 



ABSTRACT 



With greater demands being placed on postsecondary education 
as employers report that graduates have inadequate skills, interest has been 
generated in assessing student academic achievement in reliable and valid 
ways. A descriptive case study was used to outline the process that has been 
developed to identify and implement program outcomes for 55 programs at 
Western Wisconsin Technical College (WWTC) . For the purpose of the study, 
program outcome was defined as a culminating demonstration of learning as 
applied in the workplace, as well as what is expected of learners who 
successfully complete all the course work and learning experiences. Program 
outcomes are identified by faculty, validated by business and industry, 
communicated to students, and assessed at program completion. The study 
sought to determine whether there are common program outcomes within each 
school division and across divisions, how programs are written; the frequency 
and type of assessment measures identified within each division; and the 
percentage of outcomes achieved within each division and across all four 
divisions. The ability to demonstrate effective written and oral 
communication skills was found to be the most common program outcome within 
the programs at WWTC, and alternative assessment measures are used as well as 
paper-and-pencil tests. Additional research was recommended to determine 
degrees of alignment between faculty and employer perceptions of student 
success by reviewing division assessment results with the college's employer 
satisfaction survey results. (Contains 27 references.) (KC) 
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Abstract 

During the past several years, reports have indicated the inadequate skills 
of graduates, the changing demands of the workplace, and the declining 
competitiveness of U.S. firms in international marketplace. This has 
prompted the need to address educational preparation programs to ensure 
students have learned what is promised and that training is meeting the 
needs of business and industry. What better way to answer these 
questions than to assess student academic achievement. Assessment seeks 
to determine what students ought to know and to be able to do. There is a 
need to assess student academic achievement in reliable and valid ways. 
Career and technical education teachers are advocating the wider use of 
assessment measures. The purpose of this study is to outline the process 
that has been developed to identify and implement program outcomes for 
over 60 programs in a technical college. Data will be provided for 
Business, Family and Consumer Science, Health and Human Services, and 
Trades and Industrial Education associate degree and technical diploma 
programs. 



Literature Review 

For years postsecondary institutions have been assessing institutional effectiveness. 

These measures of institutional effectiveness have included retention rates; student, 
teacher and employer satisfaction; job placement and transfer rates; and student academic 
achievement. “In the last six years, there has been a dramatic change in the scope of 
assessing institutional effectiveness. The focus now is on the extent to which students are 
achieving the outcomes desired from college attendance” (Freidlander & MacDougall, 
1990, p. 93). The need to assess student academic achievement in reliable and valid 
ways, comes partly from policy makers, who believe it is essential to measure the degree 
to which participants have mastered the skills upon which training focuses (Stecher, 

Rahn, Ruby, Alt, Robyn, & Ward, 1997). Assessing student academic achievement seeks 
to determine what students should know and be able to do when they graduate from 
college. 
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In 1991 Bragg (1992) defined outcomes assessment as “evaluative processes that 
determine the results of education at the institution, program, and student level” (p. iv). 
Bragg (1995) conducted a study to identify outcomes assessment practices in two-year 



1 2 BEST COPY AVAILABLE 



postsecondary institutions. Outcomes assessment practices identified included academic 
achievement, transfer success, student and employer satisfaction, attainment of student 
goals, employment success, occupational skill attainment, job placement, and student 
persistence or retention. Few of the institutions who provided data were developing or 
implementing alternative assessment methods for the outcomes identified. Common 
assessment methods identified included standardized tests, transcript analysis, student 
and employer surveys, college placement data, licensure and certification exams, and 
enrollment patterns. Twenty-two percent of the participants indicated that the use of 
outcomes assessment was influenced by state agencies and accrediting organizations. 
Furthermore, the findings from Bragg’s 1992 study concluded that postsecondary 
institutions were viewed as leaders in outcomes assessment; however, common 
assessment methods were still being used. 

State government and regional accrediting agencies continue to encourage the 
implementation of outcomes assessment. Regional accrediting agencies require member 
institutions to implement outcome assessment measures (Marchese, 1 990). In 1 989, the 
North Central Association (NCA) Commission approved an assessment initiative for its 
member institutions to submit plans for assessing student learning (Lopez, 1997). 
Institutions affiliated with NCA began to develop assessment plans to assess their quality 
by directly assessing student learning. 

Other regional accrediting bodies have responded to sustained public dissatisfaction with 
higher education by increasing their emphasis on institutional integrity and accountability 
as conditions of accredited status. The Middle States Association of Colleges and 
Schools [On-Line] has a framework for outcomes assessment and is reviewed as part of 
the accreditation process. The New England Association of Schools and Colleges 
(NEAS & C) [On-Line] has ten standards. Standard Four-Programs and Instruction- 
includes planning and evaluation to enhance achievement of program objectives, that all 
existing programs include an assessment of their effectiveness and continued need. 

NEAS & C requires institutions to have a student outcome assessment plan aimed at 
assessing student achievement. Results from the plan are used for institutional 
improvement. The Southern Association of Colleges and Schools [On-Line] has 
institutional effectiveness criteria requiring member institutions to document quality and 
effectiveness by employing a comprehensive system of planning and evaluation. Each 
institution must develop guidelines to evaluate the quality of student learning. Standard 
Three of the Western Association of Schools and Colleges [On-Line] addresses 
institutional effectiveness. This accrediting standard required institutions to have (a) 
outcomes and clear documentation of their achievement, (b) planning activities to 
communicate quality assurance, and (c) review evaluation processes to determine 
ongoing utility for assessing institutional effectiveness. 

Each of these accrediting agencies has developed criterion based upon experience, 
research, and consultation with member institutions. These initiatives will place pressure 
on accredited institutions to demonstrate that students have learned what is promised or 
implied by the catalog’s descriptions of academic programs and that continual 
improvement of student learning has become an institutional priority (Lopez, 1998b). 
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As another approach to accountability, states have begun to look at performance-based 
budgeting of public institutions of higher education. Layzell (1998) defines 
performance-based budgeting as the allocation of resources based upon institution’s 
established goals and outcomes. Research studies conducted at public institutions of 
higher education (Ashworth, 1994; Burke & Serban, 1998; King, 2000; Layzell, 1998; & 
Pfeiffer, 1 998) report the ongoing interest for accountability and programmatic outcomes. 
Performance-based budgeting is one way that legislators may get the greatest possible 
return on a state’s expenditure (Ashworth, 1994). The Chronicle of Higher Education 
reports that states may soon link spending on colleges to institutional performance 
(Schmidt, 1998). 

In addition to federal and state legislation, and accrediting agencies, researchers have 
produced numerous reports that define and discuss the skills students need for the 
workforce and jobs of the future (America’s Choice: High Skills or Low Wages, 1990; 
Learning a Living, 1992; and SCANS, 1991). These reports likewise have contributed to 
the need for program outcomes and assessment measures in two-year colleges across the 
United States. “Community colleges will need to become “change-masters” to become 
more responsive to an uncertain future” (Phelan, 1997, p. 33). 

The purpose of this study was to determine one institution’s accomplishments to identify 
measurable outcomes used to assess student learning. It was felt that by identifying 
measurable outcomes to assess student learning; by identifying and using multiple 
measures to assess those outcomes; by collecting and interpreting data from those 
instruments; by disseminating the information about the results in structured feedback 
loops; and by using the information derived from the assessment of student learning to 
make pedagogical and curricular changes, the quality of student learning at Western 
Wisconsin Technical College (Western) could be greatly improved thereby enhancing the 
college’s commitment to student success. 

Conceptual Framework 

For the purpose of this study, program outcome was defined as a culminating 
demonstration of learning as applied in the workplace, and what is expected of the learner 
who successfully completes all of the course work and learning experiences as part of a 
technical program (Ruhland, Samson, Brewer, & Hague, 1 997). Program outcomes are 
identified by faculty, validated by business and industry, communicated to students, and 
assessed at program completion. Program outcomes were distinguished from general 
education outcomes for the purpose of this study. Faculty were encouraged to identify 
only those program outcomes that were related to the technical knowledge, skills, and 
attitudes required of students graduating from the program. Student outcomes were 
defined as changes that occur in individuals as a result of their participation in an 
educational experience (Bragg, 1992). 

This study is based on a conceptual framework of components for an effective 
assessment program. Lopez (1998a) contends that assessment of student academic 
achievement is key to (1) improving student learning, (2) enabling an institution to verify 
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that it is being accountable to its internal and external stakeholders, and (3) documenting 
to the general public and interested parties the value of investing in higher education. 
Lopez suggests that three levels of implementation characterize an institution’s 
assessment program: Level One: Beginning Implementation; Level Two, Some 
Implementation; and Level Three, On-Going Implementation. Each of these levels are 
described in Figure 1 and represent an institution’s developmental approach to 
implementing an assessment program, which focuses on the assessment of student 
academic achievement. 



Figure 1 . Levels of Assessment Implementation - North Central Association. 



Level One: Beginning 
Implementation of Assessment 
Programs 


Level Two: Making Progress 
in Implementing Assessment 
Programs 


Level Three: Maturing Stages 
of Continuous Improvement 


• Assessment plans 

available for a number of 
departments/programs — 
not all 


• Senior administration 
actively support 
assessment program, 
recognizes and rewards 
faculty’s efforts to 
implement it 


• Changes in pedagogy, 
curriculum, and/or 
academic support services 
made as a result of 
assessment data are 
communicated 


• Program goals and explicit 
objectives for learning 
developed for a number of 
departments/programs — 
not all 


• Results from measuring 
general education and the 
major are being collected 
and interpreted from some 
but not all programs 


• Assessment plan is 
annually reviewed and 
updated 


• Direct measures of student 
learning identified and 
data above the level of the 
classroom have been 
collected to a number of 
departments/programs — 
not all 


• Clear feedback loops are 
being developed to ensure 
that the results of 
assessment are used to 
improve student learning 
and teaching 


• Results of assessment of 
student learning are linked 
to academic program 
review 



Identification of program outcomes supports overall program effectiveness and 
documents student academic achievement. Assessing program outcomes can determine 
whether the student can perform the intended program outcome. Outcomes provide 
increased accountability and document student academic achievement. The benefits of 
program outcomes include (a) a clear understanding of what students are to know and be 
able to do at the completion of their program, (b) documentation of skill attainment for 
employers, (c) industry- validated outcomes as a basis for curriculum development, and 
(d) occupational skills standards for a competitive workforce. 

Methodology 

The research methodology used in this study was descriptive case study. The associate 
degree and technical diploma programs at Western were analyzed and reviewed for this 
study. Western is one of sixteen publicly-funded technical colleges in the Wisconsin 
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Technical College System. The college offers 37 associate of applied science degree 
programs; 1 8 technical diploma programs; 4 certificates; and 7 special certificate 
programs. Western serves approximately 3500 full-time equivalent students and employs 
approximately 250 faculty. 

Programs were representative of the four divisions and included 1 5 Business programs, 5 
Family and Consumer Science programs, 18 Health and Human Services programs, and 
17 Trades and Industrial Education programs. Faculty from each of the program areas 
worked as teams to identify the program outcomes, assessment measures, criteria, 
assessed if the outcomes were achieved, and provided feedback as to program 
improvement and changes as a result of the data. 

The Student Academic Achievement Assessment document (see Figure 2) was analyzed 
to collect data for this descriptive case study. This document provides qualitative and 
quantitative data related to program outcomes, assessment measures, criteria, and 
assessment feedback 

Figure 2 . Student Academic Achievement Assessment Form. 



Student Academic Achievement Assessment 
Institutional Effectiveness Indicator: (Student Achievement and Satisfaction) 



Student Knowledge and Skills at Exit 



Division: 
Graduation Date: 




Program: 

Number of Graduates: 




Program 

Outcomes 


Assessment 

Measure(s) 


Criteria 


Where 

Measured 


Data 

Did the majority of 
graduates achieve this 
outcome according to 
the established criteria? 
YES NO 


Feedback 

What 

improvements / 
changes will you 
make based on 
data? 



The four primary research questions addressed in this study were as follows: 

1 . Are there common program outcomes identified within each division? Across all 
four divisions? 

2. What percentage of program outcomes used verbs written at the upper level in each 
educational domain (cognitive, affective, and psychomotor)? 

3. What was the frequency and type of assessment measures identified within each 
division? 

cross all four divisions? 

4. What is the percentage of outcomes achieved within each division? Across all four 
divisions? 
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During the fall 1998, the college took steps to fully implement the NCA assessment plan. 
The fall in-service program began the process to assist the 55 programs within the college 
with identifying program outcomes. When writing the program outcomes, faculty were 
encouraged to write the outcomes using Bloom’s taxonomy (1956) of educational 
domains. Emphasis was placed on outcomes being written at level HI or higher. Using 
action verbs from level HI or higher ensures students moving from the most basic to the 
most complex type of learner action. 

The January 1999 in-service was used to inform faculty about alternative forms of 
assessment. Alternative forms of assessment include on-the-job evaluations, interviews, 
portfolio, paper-and-pencil tests, simulations, and interviews (Darling-Hammonds, Wise, 
& Kline, 1995). Program faculty were given until May 1999 to identify the assessment 
measure(s) for each program outcome. Following this step, faculty teams then identified 
the criteria that would measure each outcome. The last step in the assessment plan was to 
gather data from May 2000 graduates and subsequently have faculty teams assess student 
academic achievement of the program outcomes. Data collection focused on asking the 
faculty to indicate with a yes or no if they felt the majority of their graduates had 
achieved the program outcomes. This step supports the learning paradigm in which 
institutions measure effectiveness based upon student learning outcomes. In a learning 
paradigm “colleges will be concerned about the quality of exiting students and how much 
the students have learned” (Boggs, 1995-96, p. 26). 

Qualitative methods were used to analyze the data to include patterns and themes 
(commonality among program outcomes within each program and divisions), clustering 
(types of assessment measures identified and criteria used to evaluate), counting (average 
number of program outcomes for associate degree and technical diploma programs, and 
average number of criteria met) (Miles & Huberman, 1994). 

Results 

Research question 1 asked, “Are there common program outcomes identified within each 
division? Across all four divisions?” The 55 programs at Western had 12 common 
program outcomes. Table 1 lists the program outcomes in rank order by frequency. 
Demonstrate (utilize) effective communication (oral and written) skills were the most 
common program outcome identified by 25 (45%) programs. Some programs within 
each division selected a different verb to use with the outcome. Program outcomes 
identified by only one program are not reported. The average number of program 
outcomes by program in each division included (a) 1 5 program outcomes in the business 
division, (b) 16 program outcomes in the family and consumer sciences division, (c) 10 
program outcomes in the health and human services division, and (d) 1 1 program 
outcomes in the trade and industrial education division. 

Research question 2 asked, “What percentage of program outcomes used verbs written at 
the upper level in each educational domain (cognitive, affective and psychomotor)?” The 
analysis of verbs used in each outcome was coded using Bloom’s (1956) Taxonomy of 
Educational Objectives. Table 2 reveals that all program outcomes were at a Level HI or 
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Table 1 

Common Program Outcomes 



Program Outcomes 


B 


FCS 


HHS 


TI 


College 


n 


n 


n 


n 


N 


Demonstrate (utilize) effective communication 
(oral and written) skills. 


11 


5 


7 


2 


25 


Demonstrate (apply) use of computer tools, 
software, and appropriate applications. 


10 


2 


2 


1 


15 


Apply legal and ethical principles to personal, 
social, and professional behavior. 


10 


2 


2 


1 


15 


Function (operate) as a team member. 


7 


0 


2 


2 


11 


Establish (maintain) a safe work environment, 
adhere to safety procedures. 


0 


0 


3 


8 


11 


Think critically in solving problems and 
applying knowledge 


4 


0 


3 


2 


9 


Exhibit (display) professionalism. 


0 


0 


2 


6 


8 


Formulate (create) a professional development 
plan. 


5 


1 


1 


0 


7 


Apply (demonstrate) mathematical skills. 


1 


0 


0 


4 


5 


Demonstrate effective presentation skills. 


4 


0 


0 


0 


4 


Apply effective leadership skills. 


4 


0 


0 


0 


4 


Use time management techniques effectively. 


3 


0 


0 


0 


3 



Note . B = Business, FCS = Family and Consumer Sciences, HHS = Health and Human 
Services, and TI= Trade and Industrial Education. Verb changes are note in parentheses. 



higher and that 288 (42%) of the 691 outcomes were at Level V or higher. The majority 
of program outcomes within the college and four divisions were written in the cognitive 
domain at the application level. 

Research question 3 asked, “What was the frequency and type of assessment measures 
identified within each division? Across all four divisions?” Responses to types of 
assessment measures identified varied within the four divisions and across the 55 
programs. The most common assessment measure identified by all four divisions was 
performance tasks and student exhibition (see Table 3). Thirty-two (58%) of the 55 
programs identified this assessment measure within their program. In most cases 
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performance tasks and student exhibition was used for multiple program outcomes within 
an overall program. The most common assessment measure by division was “checklist 
(lab/performance)” for 10 (67%) business programs, “checklist (lab/performance)” for 
three (60%) of the family and consumer sciences programs, “professional association/ 
licensure exams” for ten (56%) of health programs, and “performance tasks and student 
exhibition” for 14 (82%) of the trades and industrial education programs. 

Table 2 

Program Outcome by Division and Domain Level 



Domain 


B 


FCS 


HHS 


TI 


College 




n 


n 


n 


n 


n 


Cognitive 












Application (C-HT) 


64 


18 


62 


29 


173 


Analysis (C-IV) 


9 


5 


10 


10 


34 


Synthesis (C-V) 


48 


5 


13 


12 


78 


Evaluation (C-IV) 


29 


9 


12 


23 


74 


Division Total 


150 


37 


97 


15 


359 


Affective 












Valuing (A-III) 


28 


13 


22 


12 


53 


Organization (A-IV) 


14 


2 


18 


17 


51 


Characterization (A-V) 


8 


5 


13 


16 


42 


Division Total 


28 


20 


53 


45 


146 


Psvchomotor 












Guided Response (P-IH) 


1 


2 


1 


2 


7 


Mechanism (PTV) 


19 


14 


17 


35 


85 


Complex Overt Response (P-V) 


25 


2 


24 


29 


80 


Adaptation (P-VI) 


0 


0 


1 


0 


1 


Origination (P-VD) 


7 


4 


0 


2 


13 


Division Total 


52 


22 


33 


68 


186 



Note . B = Business, FCS = Family and Consumer Sciences, HHS = Health and Human 
Services, and TI = Trade and Industrial Education. 



Research question 4 asked, “What is the percentage of outcomes achieved within each 
division? Across all four divisions?” Three of the 55 programs did not provide data 
related to the program outcomes achieved by the graduates. Within the business division 
programs, 226 (94%) program outcomes were achieved by 252 program graduates. The 
family and consumer sciences division had 62 (87%) program outcomes achieved by 27 



Table 3 

Type of Assessment Measures 



Division 

Assessment Measure 


B 

n 


FCS 

n 


HHS 

n 


TI 

n 


College 

n 


Accreditation exam 


0 


0 


1 


0 


1 


Capstone/clinical experience 


2 


1 


5 


6 


14 


Case analysis/presentations 


6 


0 


1 


0 


7 


Checklist (lab/performance) 


10 


3 


4 


3 


20 


Clinical evaluation forms 


0 


0 


7 


0 


7 


Competency performance sheet 


0 


0 


5 


2 


7 


Employer survey 


0 


0 


3 


0 


3 


Graduate follow-up surveys 


0 


0 


5 


0 


5 


Instructor observation 


6 


1 


5 


4 


16 


Internship 


1 


1 


0 


1 


3 


Journals 


1 


0 


1 


0 


2 


Paper-and-pencil test (written exam) 


8 


2 


5 


3 


18 


Peer/self evaluation 


1 


0 


1 


0 


2 


Performance tasks and student exhibition 


8 


3 


8 


14 


32 


Personal interviews 


0 


0 


1 


0 


1 


Portfolio 


8 


2 


6 


6 


22 


Practicum teaching 


0 


2 


0 


0 


2 


Professional association/licensure exam 


0 


0 


10 


3 


13 


Program exit exam 


0 


0 


0 


1 


1 


Simulation (project) 


6 


1 


0 


1 


8 


Student completion of program requirements 


0 


0 


1 


0 


1 


Technical report/project (written) 


7 


2 


1 


0 


10 



Note . Total programs in each division include Business (B) n = 15; Family and Consumer 
Sciences (FCS) n = 5; Health and Human Services (HHS) n = 18; and Trade and Industrial 
Education (TI) n = 17. 



program graduates. Within the health and human services division, 659 (97%) program 
outcomes were achieved by 465 program graduates. Trade and industrial education had 
183 (97%) program outcomes achieved by 216 program graduates. Overall, 625 (95%) 
of the program outcomes were achieved by 960 graduates within the four divisions. 

Discussion and Recommendations 

This study supports the efforts required of accrediting agencies (Marchese, 1990) to 
implement outcomes assessment measures. In addition, findings from this research study 
support three of the five components that NCA (Lopez, 1998a) identified as essential for 
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an assessment program. Results from Western’s assessment efforts over the past two 
years identify assessment of student academic achievement at Level Two (Making 
Progress in Implementing Assessment Programs). Faculty identified measurable program 
outcomes, alternative assessment measures, and evaluated the achievement of student 
outcomes. Continued assessment efforts will move the college to Level Three (Maturing 
Stages of Continuous Improvement) resulting in changes in pedagogy and curriculum, 
and linking the assessment results to academic program review. 

Outcomes assessment should be a central part of a teacher’s professional ethics. 
Outcomes assessment requires more than faculty time and additional committee work. It 
also involves structural changes in American higher education and the professional self- 
understanding of faculty (Holyer, 1998). When data have been analyzed for all 
programs, the next step will be to create benchmarks for comparison purposes. It will be 
imperative to provide documentation of program improvements and the resources needed 
to continue this effort. 

The ability to demonstrate effective written and oral communication skills was the most 
common program outcome within the 55 programs at Western. This finding is consistent 
with the national reports identifying the skills students’ need (America’s Choice: High 
Skills or Low Wages, 1990; Learning a Living, 1992; and Scans, 1991). Of the 12 
common program outcomes identified, four of the program outcomes align with the 
SCANS foundations (basic skills, thinking skills, and personal qualities) and four 
outcomes align with the competencies (manage resources, information, and interpersonal 
skills). 

Findings from this study indicate that faculty are moving towards implementing 
alternative assessment measures to assess student academic achievement, but common 
methods consistent with Bragg’s (1995) study were also identified in this study. Paper- 
and-pencil tests (33%) were still used by the faculty in this study. However, alternative 
assessment measures are being used by faculty to assess student academic achievement. 
When using alternative assessment measures, faculty can directly examine and judge the 
student’s actual performance on significant and relevant tasks. 

Further research should be conducted to review, by division, the feedback received 
related to the achievement of program outcomes. In addition, input from faculty is 
needed to determine what pedagogical and curriculum changes have been implemented 
as a result of the assessment of student learning. Information should be synthesized to 
note any patterns and themes faculty encountered which would assist in developing a 
feedback plan for improvement. 

Additional research should be conducted to compare the assessment results with the 
college's employer satisfaction survey results to determine degrees of alignment between 
faculty and employer perceptions of student success. In addition, program advisory 
committee members should be surveyed to determine their perception about students 
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achieving the program outcomes. Research needs to be conducted to collect data from 
other technical colleges to compare and contrast program outcomes and assessment 
measures by division and across the college. 
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